Intelligent game agents

The topic of getting machines to play classical board games at the same level as top ranking human players, is a research area of artificial intelligence that has received a considerably amount of attention from researchers during the years. But although the area to this point has received quite a lot of attention, and there exist examples of computers that has defeated top ranking human players, it is still an area that intrigues researches, and where much research still can be done. This paper presents a new way on how computers can be programmed to learn to play classical board games against humans. The model is identified as goal directed learning. The model has been implemented and tested in the domain of tic-tac-toe where it has shown to be highly successful. Suggestions on how to further improve the model such that it can be applied to more complex game domains is also presented. During the next and final 10 semester, work will continue and hopefully prove that the model can be successful integrated in a more complex game domain. An important aspect to the dissertation is existing research in the areas of artificial intelligence, game theory and previously works done in relation to getting computers to play classical board games. The first half of the paper will therefore focus on these areas. This research has been done, as investigation of these areas was considered to be of high importance before work on a personal approach could begin. Intelligent game agents II Structure of paper Besides some general introduction presented on this and previous pages, the structure of the paper is based on eight main chapters not including references and appendix. Chapter one is an introduction chapter defining the project. Chapter two through four will cover existing research and theories related to artificial intelligence and games. Chapter five through seven will cover the development of a solution model useful for learning computer to play the classical board game tic-tac-toe. Chapter eight will conclude on the dissertation. In chapter nine, a list of references can be found. Throughout the paper references to one of these references will be done by the syntax [x] where x has been replaced with a number identifying the exact reference. At the end of the paper, chapter ten will hold the appendix of the paper. Besides a section describing content of the enclosed CD, rules of chess, checkers and backgammon can be found. This is all games that are mentioned throughout the paper. Intelligent game agents III Content Preface ...................................................................................................................................................... I Abstract .................................................................................................................................................... I Structure of paper................................................................................................................................... II Content ................................................................................................................................................... III 1. Problem definition.......................................................................................................................... 1 1.1 Limitations ............................................................................................................................... 1 1.2 Purpose and goals................................................................................................................... 2 1.3 The process............................................................................................................................. 2 2. Artificial intelligence ...................................................................................................................... 4 2.1 Defining artificial intelligence ................................................................................................... 4 2.2 Searching ................................................................................................................................ 6 2.3 Fuzzy logic............................................................................................................................... 7 2.4 Fuzzy controllers ................................................................................................................... 14 2.5 Genetic algorithms................................................................................................................. 19 2.6 Artificial neural nets ............................................................................................................... 21 2.7 Bayesian networks ................................................................................................................ 25 2.8 Conclusion on artificial intelligence ....................................................................................... 32 3. Game theory ................................................................................................................................. 33 3.1 Classification of games.......................................................................................................... 33 3.2 Game trees............................................................................................................................ 36 3.3 Agents ................................................................................................................................... 41 3.4 Conclusion on game theory................................................................................................... 42 4. Related work................................................................................................................................. 43 4.1 Deep Blue.............................................................................................................................. 43 4.2 Chinook ................................................................................................................................. 44 4.3 Evolutionary Checkers........................................................................................................... 45 4.4 TD-Gammon.......................................................................................................................... 46 4.5 Conclusion on related work ................................................................................................... 47 5. Analysis......................................................................................................................................... 48 5.1 Intelligence ............................................................................................................................ 48 5.2 Requirements ........................................................................................................................ 51 5.3 Domain .................................................................................................................................. 52 5.4 Agent by classical AI ............................................................................................................. 53 5.5 A thesis for learning............................................................................................................... 56 5.6 Goal directed learning ........................................................................................................... 59 5.7 Solution complexity................................................................................................................ 64 5.8 Conclusion on analysis.......................................................................................................... 67 6. Prototype implementation........................................................................................................... 68 6.1 Design criteria........................................................................................................................ 68 6.2 Architecture ........................................................................................................................... 69 6.3 Program................................................................................................................................. 77 6.4 Conclusion on prototype implementation .............................................................................. 79 7. Models, test and evaluation ........................................................................................................ 80 7.1 Neural network based agent ................................................................................................. 80 Intelligent game agents IV 7.2 Initial solution model .............................................................................................................. 82 7.3 Improved solution models ..................................................................................................... 85 7.4 Final solution model............................................................................................................... 88 7.5 Future work............................................................................................................................ 94 7.6 Conclusion on models, test and evaluation........................................................................... 96 8. Conclusion.................................................................................................................................... 97 9. References.................................................................................................................................... 98 10. Appendix................................................................................................................................. 100 10.1 Content of enclosed CD ...................................................................................................... 100 10.2 Game Rules......................................................................................................................... 101 Intelligent game agents 1 1. Problem definition The object of this project is to investigate ways to model artificial intelligence (AI). The selected problem domain is to be a gaming environment where the problem is to develop a computer opponent. Computer games make an excellent environment for AI research. Games offer a combination of simulation and reality. The environment in which a computer player interacts is virtual, but the environment is not a simulation of a problem domain, it is the problem domain. As a result issues such as noisy sensor data, hardware dependencies and other hardware related problems can be ignored, while still addressing realistic problems. Many computer opponents found in games take a very non human like approach when playing. The follow some rules designed end implemented by the programmer and are at best on the same level as average skilled players. Further more they are unable to learn and adapt to new strategies. So is it possible, and how is it possible to create computer agent with the ability to learn and adapt strategies? 1.1 Limitations Two perspectives can be taken when considering playing games on computers. First perspective is to make a computer play classical games that has been played for generations before the invention of the computer. This includes board and card games. The other perspective is the true computer games that have become a major industry over the last years. Some examples are first person shooters and real time strategic games. Throughout this dissertation the perspective will be on classical games. This is because these games offer the best environment for focusing on AI alone. For successfully playing true computer games, it is often necessary to possess a range of other skills rather than intelligence alone. This includes coordination, precision, reflexes and speed. However if intelligence is designed correctly, there should not be any reason for, why intelligent strategies working within the domain of classical games, should not have the ability to be applied to the true computer games and other domains as well. Throughout this paper classical games will in most cases be referred to as just games. While the term true computer games will be used to define these games if necessary. Intelligent game agents 2 1.2 Purpose and goals To have a reference point throughout the dissertation, purpose and goals for the dissertation are defined. Purpose • Demonstrate an understanding of current research in the field of artificial intelligence. • Analyze and define requirements necessary to develop an intelligent agent that plays classical computer games against humans. • Contribute to the area of artificial intelligence and games with new thought, ideas and methods. Goals • Implementation of an intelligent agent in a game domain. • This paper documenting the above purposes and goal as well as phases, aspects and considerations of the project. A more in depth overview of the exact objectives of the dissertation are given in the next section. 1.3 The process This dissertation is to be separated into two parts, one for each of the two semesters 9 and 10 of Master of Science – Software Engineering. The end of the 9 semester (this semester) can be considered a major milestone with a middle exam of the dissertation. The 9 semester are to take a more theoretically approach focusing a great deal on existing research, while the 10 semester are to take a more constructional approach focusing more an a personal solution model identified at the end of the 9 semester. Object of the 9 semester involves the study of artificial intelligence, game theory and related work. This is to lay a foundation and helping when defining a solution model for an agent capable of playing the simple game of tic-tac-toe. The identified solution is then to be implemented as a prototype agent and evaluated. Experience obtained from the 9 semester is then to help on the 10 semester when a detailed system design are to be done, and the identified solution model is to be tested in a more complex game domain (probably backgammon). Extensions capable of improving the basic model identified on the 9 semester are also to be identified along with solution models for the identified extensions. Experiments and final evaluation are to be done at the end of the 10 semester. Objects of 9 semester: • Study artificial intelligence • Study game theory Intelligent game agents 3 • Study related work • Define requirements • Identify a basic solution model • Design and implementation of prototype agent capable of playing tic-tac-toe • Prototype evaluation Objects of 10 semester (preliminary): • Identify extensions and improvements to solution model • Detailed design • Agent implementation • Agent integration in game domain considered to be of high complexity • Experiment and testing • Evaluation Intelligent game agents 4 2. Artificial intelligence This chapter is to give an overview of the area of artificial intelligence. Its intention is to give the author of this report insight into this field of artificial intelligence, first and foremost to get an understanding of what artificial intelligence is, but also to gather inspiration from the field. For the reader of this report it is to work as an introduction to the field of artificial intelligence. This chapter consists of the following main sections: • Defining artificial intelligence • Searching • Fuzzy logic • Fuzzy controllers • Genetic algorithms • Artificial neural nets • Bayesian networks • Conclusion on artificial intelligence 2.1 Defining artificial intelligence Most people have a general idea about what artificial intelligence is. Asking someone and the answer will almost certainly be something like: “it is making machines or computers think”. According to Dictionary.com this answer should be rather good as the two words are defined as: Artificial Intelligence 1. Made by humans; produced rather than natural. 2. Brought about or caused by sociopolitical or other humangenerated forces or influences: set up artificial barriers against women and minorities; an artificial economic boom. 1. The capacity to acquire and apply knowledge. 2. The faculty of thought and reason. 3. Superior powers of mind. See Synonyms at mind. So the study of artificial intelligence is about producing some human made device, with the ability to acquire knowledge, and using this knowledge to reason rational. But instead of just accepting this rather loose definition, lets take a more in depth look at some of the definitions of artificial intelligence, in the hope that a more exact definition can be establish. Artificial intelligence goes back to 1950 where Allan Turing wrote a paper [5] in which he stated the question “can machines think?” In this paper he proposes a game that can test if a machine is intelligent. Today this test is known as the Turing test. Intelligent game agents 5 The proposed game in its original form is known as the imitation game. In this game there are three people, one male, one female and one interrogator. The interrogator is isolated from the male and female and all communication happens in written form. The goal for the interrogator is then to distinguish the male from the female simply by asking questions, while those being questioned should try to fool the interrogator. The Turing test is based on replacing one of participants with a computer. The goal for the interrogator is then to distinguish the human from the computer. If the computer can fool the interrogator, then it is said to be intelligent. For the Turing test to be successful the computer would need to poses the following capabilities: • Natural language processing • Knowledge representation • Reasoning abilities • Machine learning Some has argued that the Turing test is a good test because to pass it, a wide range of knowledge about the world is needed. Further more a great deal of flexibility in dealing with new situations is needed as well. However it has also been criticized for only taking a behavioral rather then psychological view of intelligence. The computer does in other words not need to be intelligent; it just has to give an impression of having it. The reason why people still debate the validity of the Turing test is most likely because a broader definition of what exactly artificial intelligence is, still have not been found. In [4] the author has collected eight definitions made from other authors, and made an interesting observation. That is, the definition of artificial intelligence can be organized into four categories. The four categories are: Systems that think like humans Systems that think rationally Systems that act like humans Systems that act rationally Table 1: Four main categories for the definition of AI First comparing the two categories at the top against the two categories at the bottom the difference is that in the topmost, the definition of artificial intelligence deals with thoughts and reasoning. Artificial intelligence should actually be able to think, whereas the goal in the two categories at the bottom is to make a system that behaves as if it is intelligent, although it might have no intelligence at all. Now comparing the left column against the right one, success in the left column is measured in human performance. While in the right column success is measured against an ideal concept of intelligence, which the author has called rationality. This deviation comes from the fact that human sometimes do irrational actions. The meaning of irrational action in this context is simply the fact that humans sometimes are making mistakes unintentionally, or (having the power to) acting against all logic in a given situation. An exact definition of artificial intelligence does unfortunately not exist. Although the term is widely used, it does not attempt to give a single and universally acceptable Intelligent game agents 6 definition. Almost every book on the subject of artificial intelligence, seem to have its own definition. The problem is then to find the definition that appeal to one. The following definition is made by Amit Konar in [1]. Artificial intelligence can be defined as the simulation of human intelligence on a machine, so as to make the machine efficient to identify and use the right piece of “knowledge”. Which in my opinion is in the same category as the Turing test, and that is systems that act like humans. Whether one agree with this definition or not, it should be intuitively clear, that artificial intelligence is about getting machines to find solutions to problems in ways somewhat equal to that of humans, and this involves borrowing characteristics from human intelligence, and applying them to machines. 2.2 Searching Search approaches are described in [1]. One way of looking at artificial intelligence, is as search. One has a set of examples and a set of rules. The learning process is simply to apply the set of rules to the examples and start looking for a solution. In many cases the task of AI can simply be reduced to the problem of performing a search within a given search space. The search space is the collection of all possible and reachable solutions, and the search for a solution is then the problem solving. 2.2.1 Generate and test The most straightforward approach is simply to test every possible solution to a problem and then picking the best one. However as all but the simplest problem either has an infinite, or finite but very large search space, this approach is impractical. It will simply be impossible to search every solution as it will take to long or even forever. One common use of this approach is in game playing programs. The computer will try to look as far ahead in the game as possible, and the consequently selecting the best move. This form of search is known as game trees, and is explained in section 3.2. For searching large or infinite search spaces, the simplest alternative is to do a random search. Random solutions are generated until a satisfying solution is found or a sufficient large search space has been tested. There is no guaranty that this approach will find a useful solution unless allowed to run in an infinite time. If the search space contains very few non bad solutions, random search are unlikely to be of any use. 2.2.2 Hill climbing This approach starts by generating a random solution which is to act as the starting state. But in contrast to the random search this solution is used to generate one or more close variants. If a better variant is found this is instead used to guide the further search and so on. If after several iterations no improvement is found, the search terminates. However in the case that all neighborhood states are equal or worse than the current best solution, and the best solution is not a satisfying goal, then the search Intelligent game agents 7 algorithm is said to be trapped at a hillock or local extrema. One way to overcome this is by randomly selecting a new starting state, and then continuing the search process. 2.2.3 Heuristic search Heuristic means rule of thumb. The general approach in heuristic search is to use one or more heuristic functions to determine the best candidate among a collection of legal states that can be generated from the current state. The difficult task in heuristic search, is selecting a satisfying heuristic function, as the heuristic function has to be intuitively selected or created. 2.2.4 Means and ends analysis This search method attempts to reduce the gab between the current state and the goal state. One common way of doing this is by measuring the distance between the current state and the goal state, and then applying some operator that will reduce the gap. In mathematical theorem-proving, means and ends analysis is often used. 2.3 Fuzzy logic Fuzzy logic [1], [16], [19] and [20], deals with the area of artificial intelligence known as uncertainty handling. The primary point in fuzzy logic is to define a way to deal with imprecision. The point is that in real life problems the answer to a question is only rarely of the form true or false. Fuzzy logic is an extension to traditional two valued logic. Traditional two valued logic uses the binary pair {0, 1} to represent the two truth values false and true. It is however argued by Lotfi Zadeh, the father of fuzzy logic, that many real world sets of today are defined by non sharp boundaries. Example of such a set could be height. This is because height of an object often simply cannot be answered as either being high (true) or not high (false). Instead an object can be high to a degree. Fuzzy logic uses the whole interval between [0, 1]. This gives the ability to model a continuous change from false to truth and thereby giving someone ability to be somewhat high instead of either just high or not high. 2.3.1 Fuzzy sets A set is a collection or class of objects that shares a property. A set is specified by its members (items or elements). An example could be the set of integers greater than 0 and smaller then 5 specified by A = {1, 2, 3, 4}. In a fuzzy set each member has a membership degree between 0 and 1. The higher the number is, the higher the membership is. The elements of a fuzzy set are taken from a universe. The universe holds all the elements that come into consideration when determine the fuzzy sets members. Every element in the universe is member of the fuzzy set to some degree. This membership degree could be zero. The set of elements that have a non-zero membership is called the support. The function that ties each element from a universe to a fuzzy set is called a membership function and is often denoted by the letter μ. Figure 1 is an illustration of a membership function that defines to what degree someone is tall. Intelligent game agents